Building a Transparent Batching Layer for Storm

 Building a Transparent Batching Layer for Storm Matthias J. Sax, Malu Castellanos HP Laboratories HPL-2013-69 streaming data, distributed streaming system, batching, performance, optimization Storm is a distributed intra-node-parallel stream processing system built for very low latency processing. One major drawback of Storm is its relatively low throughput. In order to increase Storm's throughput, we designed a batching layer for Storm that is able to improve Storm's throughput significantly. In order to get a high user acceptance, we did not modify Storm but build the batching layer "on top" of it. The layer is transparent to the Storm system as well as to the user code, i.e., the user-defined functions. Thus, already developed Storm programs (so-called topologies) can benefit from our batching layer without modification. In this document, we describe the design of the batching layer and provide inside into some implementation details. External Posting Date: July 6, 2014 [Fulltext] Approved for External Publication Internal Posting Date: July 6, 2014 [Fulltext] Copyright 2014 Hewlett-Packard Development Company, L.P. Hewlett-Packard Laboratories 1501 Page Mill Road, Palo Alto (CA), USA Building a Transparent Batching Layer for Storm HP Labs Technical Report HPL-2013-69 Matthias J. Sax Databases and Information Systems Group Humboldt-Universität zu Berlin, Germany [email protected] www.dbis.informatik.hu-berlin.de Malu Castellanos Hewlett-Packard Laboratories Palo Alto (CA), USA [email protected] www.hpl.hp.com